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Abstract 


An  international  computational  aerodynamics  study  under  the  auspices  of 
The  Technical  Cooperation  Program  (TTCP)  Weapons  Technology  Panel  2 
(APN-TP-2)  involving  participants  from  defense  research  laboratories  of  the 
United  States,  United  Kingdom,  Canada,  and  Australia  was  recently  completed. 
The  purpose  of  this  study  was  to  examine  computational  predictive  technologies 
for  finned  missile  shapes  by  comparing  Navier-Stokes  predictions  to 
experimental  data.  Experimental  data  consisting  of  surface  pressures  on  the 
body  and  fins,  flow  field  pitot  pressures,  and  force  measurements  were  available 
for  comparison  to  the  computational  results.  The  computational  results  for  this 
study  established  an  extensive  database  for  evaluation  and  comparison.  The  full 
database  consists  of  results  from  six  Navier-Stokes  codes  obtained  by  seven 
multi-block  patched  and  unstructured  grids  for  five  distinct  test  cases.  The 
statistical  analysis  techniques  developed  to  help  provide  an  evaluation  of  the 
predictive  techniques  are  described.  Quantitative  results  of  the  analysis  of  the 
differences  between  computational  and  experimental  results  are  presented 
graphically  and  quantitatively  in  terms  of  medians,  standard  deviation,  and  a 
figure  of  merit  to  assist  in  the  overall  evaluation  of  the  study  results.  The  good 
performance  achieved  using  the  Spalart-Allmaras  turbulence  model  and  multi¬ 
block  patched  and  unstructured  grid  techniques  are  noted  in  the  findings. 
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Abstract 

An  international  computational  aerodynamics  study 
under  the  auspices  of  The  Technical  Cooperation 
Program  (TTCP)  Weapons  Technology  Panel  2 
(WPN-TP-2)  involving  participants  from  defense 
research  laboratories  of  the  United  States,  United 
Kingdom,  Canada,  and  Australia  was  recently 
completed.  The  purpose  of  this  study  was  to  examine 
computational  predictive  technologies  for  finned 
missile  shapes  by  comparing  Navier-Stokes 
predictions  to  experimental  data.  Experimental  data 
consisting  of  surface  pressures  on  the  body  and  fins, 
flow  field  pitot  pressures,  and  force  measurements 
were  available  for  comparison  to  the  computational 
results.  The  computational  results  for  this  study 
established  an  extensive  database  for  evaluation  and 
comparison.  The  full  database  consists  of  results 
from  six  Navier-Stokes  codes  obtained  by  seven 
participants,  seven  turbulence  models,  and  structured, 
multi-block  patched  and  unstructured  grids  for  five 
distinct  test  cases.  The  statistical  analysis  techniques 
developed  to  help  provide  an  evaluation  of  the 
predictive  techniques  are  described.  Quantitative 
results  of  the  analysis  of  the  differences  between 
computational  and  experimental  results  are  presented 
graphically  and  quantitatively  in  terms  of  medians, 
standard  deviation,  and  a  figure  of  merit  to  assist  in 
the  overall  evaluation  of  the  study  results.  The  good 
performance  achieved  using  the  Spalart-Allmaras 
turbulence  model  and  multi-block  patched  and 
unstructured  grid  techniques  are  noted  in  the  findings. 

Introduction 

An  international  computational  aerodynamics  study 
was  recently  completed  under  the  auspices  of  The 
Technical  Cooperation  Program  (TTCP)  Weapons 
Technology  Panel  2  (WPN-TP-2).  It  involved 
participants  from  defense  research  laboratories  of  the 
United  States,  United  Kingdom,  Canada,  and 
Australia  was  recently  completed.  This  current  study 
follows  a  previous  study  on  an  ogive-cylinder 
body1 2  3  which  was  reported  earlier.  The  purpose  of 
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this  study  was  to  examine  computational  predictive 
technologies  for  finned  missile  shapes  by  comparison 
of  Navier-Stokes  predictions  to  experimental  data. 
Experimental  data  consisting  of  surface  pressures  on 
the  body  and  fins,  flow  field  pitot  pressures,  and  force 
measurements  were  available  for  comparison  to  the 
computational  results.  The  computational  results  for 
this  study  have  established  an  extensive  database  for 
evaluation  and  comparison.  This  database  consists  of 
results  from  six  Navier-Stokes  codes  by  seven 
participants,  seven  turbulence  models,  and  structured, 
multi-block  patched  and  unstructured  grids  for  five 
distinct  test  cases.  The  statistical  analysis  techniques 
developed  to  help  provide  an  evaluation  of  the 
predictive  techniques  are  described.  Quantitative 
results  of  the  analysis  of  the  differences  between 
computational  and  experimental  results  are  presented 
graphically  and  quantitatively  in  terms  of  medians, 
standard  deviation,  and  a  figure  of  merit  to  assist  in 
the  overall  evaluation  of  the  study  results. 

The  full  scope  of  the  computational  study  resulted  in 
a  large  number  of  test-case  results  for  evaluation  with 
respect  to  each  other  and  to  the  effects  of  grid 
resolution,  choice  of  turbulence  model,  and 
computational  technique.  For  the  purpose  of  this 
paper,  a  subset  of  the  full  study  was  selected  for 
analysis.  The  analysis  reported  here  considers 
surface  pressure  results  for  two  test  cases  for  a  finned 
missile  at  Mach  2.5  and  14°  angle  of  attack. 

Model  and  Experimental  Data 

The  experimental  model  for  the  results  considered  in 
this  paper  is  shown  in  Figure  1  mounted  in  the  wind 
tunnel  at  NASA  Langley  Research  Center.  This 
model  has  a  3-cal.  ogive  nose  followed  by  a 
cylindrical  section  of  12-cal.  length.  Four  delta  fins 
are  located  on  the  aft  portion  of  the  cylinder  starting 
at  x/d  =  6.639  and  ending  at  x/d  =  10.013.  The 
leading  edge  sweep  angle  of  the  fins  is  18.435°.  A 
computed  visualization5  of  the  flow  over 
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the  model  at  Mach  =  2.5,  alpha  =  14°  is  shown  in 
Figure  2.  Pitot  pressure  contours  are  shown  at 
selected  axial  stations.  The  substantial  flow 
separation  is  clearly  evident  in  the  developing  vortex 
structures  evolving  from  the  leading  edge  of  the  nose 
and  the  fins.  Obviously,  this  is  a  very  challenging  test 
case  for  computational  techniques. 

Experimental  data  in  the  form  of  surface  pressures, 
pitot  pressures  in  the  outer  flow  field  and  strain  gage 
force  and  moment  measurements  are  available  to 
compare  with  the  computational  results4.  The  layout 
of  the  surface  pressure  taps  on  the  cylindrical  body 
and  fins  is  shown  in  Figure  3.  During  the 
measurements,  the  model  was  rolled  in  increments  to 
permit  measurements  for  the  full  circumferential 
extent  of  the  flow  field.  For  the  analysis  discussed  in 
this  paper,  only  surface  pressure  measurements  at 
selected  locations  on  the  cylindrical  body  and  fins 
were  utilized.  The  positions  selected  for  the 
cylindrical  body  were  x/d  =  5.5,  7.927,  8.788  and 
9.937.  Referring  to  Figure  3,  the  fin  pressures 
selected  were  identified  as  rows  a,  c,  e,  and  g. 

An  example  of  calculations  for  surface  pressure 
compared  to  experimental  data5  is  shown  in  Figure  4 
at  x/d  =  5.5  for  the:Spalart-AlImaras(SA),:  Baldwin- 
Barth(BB),and:Baldwin-Lomax(BL)  turbulence 
models.  The  SA  results  pick  up  the  first  suction  peak 
very  well,  but  the  second  suction  peak  and  lee  side 
separated  flow  are  less  well  defined.  Results  for  all 
three  turbulence  models  have  good  agreement  on  the 
wind  side  and  at  the  pitch  plane  of  symmetry.  The 
difference  between  calculation  and  experiment  for 
these  same  data  is  shown  in  Figure  5.  As  expected, 
the  difference  between  calculation  and  experiment  is 
very  small  on  the  wind  side;  however,  the  difference 
is  quite  pronounced  for  the  lee  side  separated  flow. 
These  data  (difference  between  experiment  and 
computation)  are  operated  on  by  the  statistical 
analysis  technique  described  in  the  next  section  to 
develop  the  statistical  results. 

In  the  discussion  that  follows,  many  references  are 
made  to  the  various  data  sets  using  a  short  hand 
notation.  This  notation  is  shown  in  Tables  1  and  2 
where  the  CFD  codes  and  turbulence  models  used  in 
this  study  are  listed. 

Statistical  Methodology 

The  wide  availability  of  powerful  and  affordable 
computing  resources  has  impacted  the  way  in  which 
data  analysis  is  conducted  at  a  fundamental  level. 


expressions  such  as  data  mining,  exploratory  data 
analysis,  and  data  visualization  have  entered  the 
lexicon.  A  basic  tenet  is  that  data,  properly 
summarized  and  effectively  portrayed,  may  speak 
directly  to  the  subject-matter  expert  without  resort  to 
extensive  or  arcane  statistical  methodology. 

A  set  of  experimental  data  serves  as  a  baseline  for 
comparison  of  the  flowfield  predictors.  The 
technique  involves  the  comparison  of  two  data  sets:  a 
set  of  experimental  data  and  a  set  of  computed 
results.  To  facilitate  discussion,  the  experimental  data 

are  denoted  as  Y-1 . 

The  subscripts  (i,  j)  identify  the  location  of  the 
pressure  measurements  on  the  projectile. 

For  the  body,  i  =  5.5,  7.93,  8.79,  9.94  specifies  the 
axial  station  and  0  <  j  <  360°  in  the  circumferential 
position.  For  the  fins,  (i,j)  are  the  usual  Cartesian 
coordinates  (Figure  3).  The  superscripts  (k,  l) 
distinguish  subsets  of  the  experimental  data:  k  =  1 
corresponds  to  the  body  and  k  =  2  the  fins;  /  =1 
corresponds  to  0°  roll  angle  (a.k.a.  +  -  configuration) 
and  l  -  2  to  45°  roll  angle  (a.k.a.  x-configuration). 
The  corresponding  computed  values  are  denoted  as 

Xy  ,  where  the  subscripts  and  superscripts  perform 

the  same  role  for  the  experimental  data.  If  the 
difference  between  a  computed  value  of  surface 
pressure  and  the  corresponding  experimental 

measurement  is  defined  as  error,  then  E*f  -  Y~l - 

XT'  kl 

X  ij  is  the  error  at  location  (/,  j)  for  data  set  ( k ,  /). 

Reduction  of  the  subsets  of  pressure  differences 
must  be  accomplished  before  a  simultaneous 
comparison  of  flow  field  calculations  can  be 
effectively  undertaken.  Toward  this  end,  a  box  and 
whisker  plot  is  a  useful  device  for  data  visualization, 
Figure  6.  To  construct  a  box  and  whisker  plot  for  a 

i-'kl 

data  set  {  E-  } ,  the  errors  are  first  ordered  from 

smallest  to  largest.  The  bottom  and  top  of  the  box  are 
the  25th  and  75th  percentiles,  respectively,  of  the 
ranked  errors;  the  median,  or  50th  percentile,  is 
represented  by  the  line  approximately  midway  within 
the  box.  The  whiskers  are  the  lines  that  emanate  from 
the  box  and  extend  toward  the  extreme  values  of  the 
data  set.  A  value  between  1.5-  and  3h  above  the 
upper  box  limit  (75th  percentile)  is  designated  as  an 
outlier ,  and  values  in  excess  of  3h,  an  extreme  value . 
An  entirely  analogous  statement  holds  for  values 
below  the  lower  box  limit  (25th  percentile).  Figure  6 
provides  a  compact  description  of  how  the  values 
within  a  subset  are  distributed,  facilitating 
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comparison  across  several  sets  of  data.  The 
designation  of  certain  points  as  outliers  or  extreme 
values  is  an  attempt  to  maintain  graphic  fidelity  while 
avoiding  undue,  and  possibly  misleading,  extension 
of  whisker  lengths  caused  by  only  a  few  values. 
Outliers  and  extreme  values  are  usually  the  object  of 
close  scrutiny  by  data  analysts  to  verify  their 
authenticity  and  to  exclude  the  possibility  of  their 
resulting  from  some  type  of  spurious  behavior. 

If  the  computed  and  experimental  values  coincide  at 
every  location  (i,  j),  the  box  and  whisker  plot  will 
collapse  to  a  single  value-zero.  Failing  that,  a  thin 
box  with  short  whiskers  indicates  good  overall 
agreement.  For  example,  inspection  of  Fig.  7a 
suggests  that  data  sets  [SA(H)]  and  [PDT(L)]  may  be 
closest  to  this  ideal. 

The  statistical  graphics  in  Figures  7-8,  although 
enlightening,  suggestive,  and  highly  appropriate  for 
an  initial  screening,  still  do  not  provide  a  quantitative 
assessment  of  which  error  sets  represent  closest 
agreement  between  experiment  and  computation. 
Clearly,  we  would  like  the  errors  to  be  tightly 
clustered  about  zero.  Such  a  distribution  of  errors 
would  be  reflected  in  a  location  parameter  (a  mean  or 
median)  and  a  dispersion  parameter  (a  standard 
deviation  or  interquartile  range),  assuming  values 
close  to  zero,  Figure  9.  An  attempt  to  formally  rank 
the  effectiveness  of  the  computation  procedures,  as 
revealed  through  the  error  sets  should  involve  a 
statistic  that  includes  measures  of  both  location  and 
dispersion. 

For  this  purpose,  as  a  statistic  we  chose  the  distance 
between  the  point  determined  by  the  sample  median 
and  standard  deviation,  denoted  as  (m,  s),  and  the 
origin  (0,0).  Notice  that  the  origin  corresponds  to 
perfect  agreement  between  calculation  and 
experiment,  so  the  closer  (m,  s)  is  to  the  origin,  the 
better  the  agreement. 


The  distances  ym2  +  s2  were  determined  for  the 
body  +  fins  results  for  roll  -  0  and  roll  =  45°.  These 
results  are  presented  in  Tables  3  and  4  in  rank  order 
with  the  closest  agreement  listed  first. 

More  complicated  ranking  procedures  are  possible, 
but  to  engage  in  them  without  a  compelling  reason 
serves  no  practical  purpose  and  was  not  undertaken. 

The  results  of  the  ranking  procedure  are  entirely 
consistent  with  the  preliminary  graphics.  For  the 


statistician,  failure  to  account  for  error  in  the 
experimental  measurements  1-  is  troublesome,  but 

engineering  experience  leaves  the  subject-matter 
expert  convinced  that  concern  over  this  point  is 
unwarranted.  In  this  study,  only  a  single  set  of 
experimental  data  is  available.  All  of  the  flow  field 
predictors  are  deterministic,  and  the  measurement 
error,  whatever  its  magnitude,  remains  confounded 
with  the  recorded  observations. 


Discussion  of  Results 

The  results  for  roll  =  0  are  shown  in  Figures  7a,  7b, 
and  7c.  The  appearance  of  the  outliers  indicates  the 
consistency  of  the  results.  The  outliers  could  be 
caused  by  turbulence  model  effects  and/or 
deficiencies  in  the  computational  grid.  This 
information  would  provide  the  computational 
specialist  incentive  to  reevaluate  his  results  in  these 
areas.  In  considering  the  variety  of  turbulence 
models,  CFD  codes,  and  grid  configurations,  it  is 
informative  to  consider  the  results  in  similar 
groupings. 

First,  consider  the  structured  grid  results  for  body 
alone.  These  results  are  represented  by  the  following 
ID’s:  SA(H),  BB(H),  BL(H),  BL10(H),  SST(H), 
PW(S),  SA(D),  BL(L),  PDT(L),  SA(TB).  One 
observation  is  that  the  results  using  the  BL  turbulence 
model  have  the  greatest  presence  of  outliers.  The 
results  using  SA  turbulence  model  appear  to  be  more 
consistent  than  others,  yielding  no  extreme  values. 

The  SST  and  PDT  turbulence  models  also  provide 
encouraging  results.  For  the  unstructured  grid  results 
[ID's:  SA(K),  SA(B)],  the  presence  of  outliers  is 
greater  than  for  the  results  using  the  same  turbulence 
model  with  structured  grids. 

The  results  for  fins  are  shown  in  Figure  7b. 
Noteworthy  here  are  the  results  for  PW(S)  and 
SA(D).  These  results  were  obtained  using  the  same 
structured  grid  and  they  have  no  extreme  values. 
These  results  are  better  than  they  achieved  for  the 
body  alone.  The  results  achieved  using  the  SA 
turbulence  model  appear  to  be  consistently  better  than 
that  for  other  turbulence  models.  One  exception  is 
the  SA(K)  result  which  has  several  extreme  values. 
Time  did  not  permit  close  examination  of  these  data 
to  ensure  that  the  data  reduction  technique  did  not 
introduce  some  errors  when  interpolating  the 
irregular,  unstructured  data  to  the  regular  grid  of  the 
experimental  data. 
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The  results  for  the  body  plus  fins  are  shown  in  Figure 
7c.  These  results  are  dominated  by  the  fins  and  are 
quite  similar  to  the  results  from  Figure  7b. 

The  results  for  roll  =  45  are  shown  in  Figures  8a,  8b, 
and  8c.  The  results  for  body  alone  are  very  similar  to 
the  results  for  roll  =  0.  Again,  the  SA  turbulence 
model  results  appear  to  be  consistently  somewhat 
better  than  results  for  other  turbulence  models.  There 
is  some  indication  that  the  fin  results  have  a  greater 
amount  of  outliers  than  the  fin  results  for  the  roll  =  0 
case.  Also,  the  PW(S)  and  SA(D)  results,  which 
were  obtained  using  similar  CFD  codes  and  the  same 
grid,  are  the  only  results  with  no  extreme  values. 

The  plots  shown  in  Figures  9a  and  9b  indicate  the 
relative  rank  ordering  of  the  results  in  terms  of  the 
median  and  standard  deviation.  The  result  closest  to 
the  origin  (0,0)  would  be  the  best.  In  Figure  9a,  for 
roll  =  0,  the  results  indicate  that  the  structured  grids 
using  SA  turbulence  model  perform  the  best,  although 
the  results  obtained  using  SST,  PDT,  and  BB 
turbulence  models  also  performed  well.  The  results 
shown  in  Figure  9b  for  the  roll  =  45  case  are  similar. 
Tables  3  and  4  sort  the  tabulated  results  for  these  two 
figures  from  best  to  worst.  The  median  values  are 
consistently  less  than  zero  indicating  that  the 
computations  are  under  predicting  the  surface 
pressures.  This  trend  is  also  observed  in  Figure  4. 

The  statistical  data  suggest  that  the  roll  =  0  case  is 
slightly  better  predicted  than  the  roll  =  45  case. 
However,  the  differences  are  so  slight  that  this  may 
not  be  statistically  significant.  For  both  sets  of  data 
(Table  3  and  Table  4),  three  of  the  top  four  results 
were  obtained  by  Haroldsen5;  his  results  were 
obtained  using  a  structured,  patched  multi-block  grid 
generation  technique. 

Concluding  Remarks 

Traditionally,  an  engineer  evaluates  flow  field 
computations  by  visual  inspection  and  a  qualitative 
comparison  with  experiment,  an  often  satisfactory 
process  for  a  small  number  of  data  sets.  However,  in 
this  study,  the  number  of  data  sets  was  sufficiently 
large  to  make  the  evaluation  and  comparison  of  the 
results  an  unwieldy  task.  Thus,  a  statistical  approach, 
with  emphasis  on  data  visualization,  was  used  to 
assist  the  engineer  in  comparing  Navier-Stokes 
predictions  to  experimental  measurements.  The 
method  was  applied  to  a  set  of  computational  results 
obtained  by  seven  participants  using  six  CFD  codes 
and  seven  turbulence  models  for  a  finned  missile  at 
Mach  2.5  and  14°  angle  of  attack.  The  analysis 
showed  that  using  more  computer-intensive 


turbulence  models  did  not  provide  correspondingly 
superior  results.  It  was  also  noted  that  the  results 
obtained  using  unstructured  grids  were  very 
competitive  with  those  obtained  using  conventional 
structured  grids.  The  statistical  procedure  developed 
for  this  study  makes  few  assumptions;  it  is  of 
particular  value  when  comparing  a  large  number  of 
data  sets  and  when  an  impartial  quantitative 
assessment  is  desired. 
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Table  1.  List  of  Navier-Stokes  Codes  Utilized 


Participant  (Ref) 

Identification 

Code  Name 

Brief  Description 

Birch  (8) 

TB 

FLUENT 

FV,  Implicit,  structured 

Bulbeck  (6) 

B 

FLUENT 

FV,  Implicit,  unstructured 

Dinavahi 

D 

OVERFLOW 

FV,  Implicit,  Upwind,  structured 

Haroldsen  (5) 

H 

WIND  1.0 

FV/FD,  Implicit,  multi-block 

Kinsey  (10) 

K 

COBALT 

FV,  Implicit,  unstructured 

Lesage  (9) 

L 

WIND  1.0 

FD,  Implicit,  structured 

Sahu  (7) 

S 

ZNSFLOW 

FD,  Implicit,  Upwind,  structured 

Table  2.  List  of  Turbulence  Models  and 
Designation  Code 


Table  3.  Ordered  Figure  of  Merit  for 
Body  +  Fins,  Roll  =  0 


L  ID 

Median 

Std  Dev 

Distance 

ism 

-0.00044 

0.013838 

0.013845 

-0.00159 

0.01411 

0.014199 

Em 

-0.00223 

0.014433 

0.014605 

■mm 

-0.00147 

0.015115 

0.015186 

-0.00096 

0.016014 

0.016042 

■mm 

-0.00182 

0.016353 

0.016454 

mi 

-0.00201 

0.016781 

0.016901 

■Hi-fl 

-0.00217 

0.017159 

0.017295 

PW(S) 

-0.00198 

0.018933 

mm 

SA(D) 

0.003066 

0.019456 

0.019696 

BL10(H) 

-0.00274 

0.021424 

0.021599 

mi 

-0.00231 

0.023618 

0.023731 

Table  4.  Ordered  Figure  of  Merit  for 
Body  +  Fins,  Roll  =  45 


Compared  to  Experimental  Data,  M=  2.5,  Alpha-  14°. 


Body  Alone,  Three  Turbulence  Models,  Mach  -  2.5,  Alpha  -  14  °. 


SA(H)  BB(H)  BL(H)  BL10(H)  SST(H)  PW(S)  SA(D)  BL(L)  PDT(L)  SA(K)  SA(B)  SA(TB) 


Figure  7a.  Box  and  Whisker  Plot,  Body  Surface  Pressures,  Roll  =  0. 


Figure  7b.  Box  and  Whisker  Plot,  Fin  Surface  Pressures,  Roll  =  0. 
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^ure  9a.  Body  plus  Fins,  Roll  =  0,  Standard  Deviation  vs  Median. 
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Figure  9b.  Body  plus  Fins,  Roll  =  45,  Standard  Deviation  vs  Median. 
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